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We show that iterative coding systems can not surpass capacity using only 
quantities which naturally appear in density evolution. Although the result in itself 
^sO \ is trivial, the method which we apply shows that in order to achieve capacity the 

various components in an iterative coding system have to be perfectly matched. 
This generalizes the perfect matching condition which was previously known for 
the case of transmission over the binary erasure channel to the general class of 
^ ' binary-input memoryless output-symmetric channels. Potential applications of this 

perfect matching condition are the construction of capacity-achieving degree dis- 
tributions and the determination of the number required iterations as a function 
of the multiplicative gap to capacity. 

IT) . 

Tj- . Assume we are transmitting over a binary-input memoryless output-symmetric (BMS) 

channel using sparse graph codes and an iterative decoder. Why can we not surpass 
capacity with such a set-up? The trivial answer is of course given by the converse to the 
channel coding theorem. In this paper we give an alternative proof of this fact which 
uses only quantities which are naturally tied to the setup of iterative coding. We show 
(j '. that in order to achieve capacity the various components in an iterative coding system 

have to be perfectly matched. This generalizes the perfect matching condition which was 
; previously known for the case of transmission over the binary erasure channel (BEC) to 



5_i ■ the general class of BMS channels. 



The first bound in which capacity was derived explicitly from density evolution was 
given by Shokrollahi and Oswald [1, 2] for the case of transmission over the BEC. For 
the same channel, a very pleasing geometric bound using ten Brink's EXIT charts ([3-6]) 
was later given by Ashikhmin, Kramer and ten Brink using the Area Theorem [7-9] . See 
also [10-13] for related work and the extension to parallel concatenation. 

For general BMS channels, this geometric interpretation is unfortunately no longer 
valid since the Area Theorem is no longer fulfilled by the component-wise EXIT curves. 
Motivated by the pleasing geometric statement observed for the BEC, a similar chart, 
called MSE chart was constructed by Bhattad and Narayanan [14]. Assuming that the 
input densities to the component codes are Gaussian, this chart again fulfills the Area 
Theorem. The introduction of this function was motivated by the elegant relationship 
between mutual information and signal-to-noise observed by Guo, Shamai and Verdu 
[15, 16]. In order to apply the MSE chart in the context of iterative coding the authors 
proposed to approximate the intermediate densities which appear in density evolution 



by "equivalent" Gaussian densities. This was an important first step in generalizing the 
matching condition to the whole class of BMS channels. 

In the following we show how to overcome the need for making the Gaussian ap- 
proximation by using generalized EXIT (GEXIT) functions [17]. The bound which we 
derive and its geometric interpretation is quite similar to the one given for the BEC: We 
represent the "actions" performed by each component code by their respective GEXIT 
functions. By construction the area under these GEXIT functions is related to the rate of 
the corresponding codes. Further, we show that if we are transmitting below the thresh- 
old of iterative coding then these two curves do not overlap. Using then an argument 
identical to the one introduced for the BEC it follows that in order to achieve capacity 
the two individual component curves have to be perfectly matched. 

There are two obvious potential applications of the perfect matching condition. First, 
assuming a perfect matching of the component curves and working backwards, one might 
be able to exhibit capacity-achieving degree distribution pairs. Secondly, the geometric 
picture given by the bound seems to provide the natural setting to prove that the num- 
ber of iterations of iterative codings systems scales at least like 0(1/5), where 5 is the 
multiplicative gap to capacity, another long standing conjecture of iterative coding. 



1 EXIT Charts and the Matching Condition for BEC 

To start, let us review the case of transmission over the BEC(h) using a degree distribution 
pair (A,p). In this case density evolution is equivalent to the EXIT chart approach and 
the condition for successful decoding under BP reads 

c( x ) = l-p(l-x)< \-\x/h) = v~\x). 
This is shown in Fig. ^for the degree distribution pair (A(x) = x 3 ,p(x) = x 4 ). The area 

h = 0.58 




x 



Figure 1: The EXIT chart method for the degree distribution (A(x) = x 3 ,p(x) = x 4 ) 
and transmission over the BEC(h = 0.58). 

under the curve c(x) equals 1 — Jp and the area to the left of the curve v^ 1 (x) is equal 
to hfX. By the previous remarks, a necessary condition for successful BP decoding is 
that these two areas do not overlap. Since the total area equals 1 we get the necessary 
condition hjA + 1 — f p < 1. Rearranging terms, this is equivalent to the condition 

l-C Sh = h< jl = i- r (X,p). 

In words, the rate r(A, p) of any LDPC ensemble which, for increasing block lengths, 
allows successful decoding over the BEC(h), can not surpass the Shannon limit 1 — h. As 



pointed out in the introduction, an argument very similar to the above was introduced by 
Shokrollahi and Oswald [1,2] (albeit not using the language and geometric interpretation 
of EXIT functions and applying a slightly different range of integration). It was the first 
bound on the performance of iterative systems in which the Shannon capacity appeared 
explicitly using only quantities of density evolution. A substantially more general version 
of this bound can be found in [7-9] (see also Forney [18]). 

Although the final result (namely that transmission above capacity is not possible) 
is trivial, the method of proof is well worth the effort since it shows how capacity enters 
in the calculation of the performance of iterative coding systems. By turning this bound 
around, we can find conditions under which iterative systems achieve capacity: In par- 
ticular it shows that the two component-wise EXIT curves have to be matched perfectly. 
Indeed, all currently known capacity achieving degree-distributions for the BEC can be 
derived by starting with this perfect matching condition and working backwards. 

2 GEXIT Charts and the Matching Condition for 
BMS Channels 

Let us now derive the equivalent result for general BMS channels. As a first ingredient 
we show how to interpolate the sequence of densities which we get from density evolution 
so as to form a complete family of densities. 

Definition 1 (Interpolating Channel Families) Consider a degree distribution pair 
(A,p) and transmission over the BMS channel characterized by its L-density c. Let 
a_i = A anda = c and set a aj a G [—1,0] , to a a = — «a_! + (l+o;)ao. The interpolating 
density evolution families {a Q ,}^ = „ 1 and {b a }^l are then defined as follows: 

K = ^pi*®-i l \ a > 0, 

i 

a^^A^b*^, a>0, 

i 

where * denotes the standard convolution of densities and alb denotes the density at 
the output of a check node, assuming that the input densities are a and b, respectively. 

Discussion: First note that a^ (b^), £ G N, represents the sequence of L-densities of 
density evolution emitted by the variable (check) nodes in the ^-th iteration. By starting 
density evolution not only with ao = c but with all possible convex combinations of 
A and c, this discrete sequence of densities is completed to form a continuous family 
of densities ordered by physical degradation. The fact that the densities are ordered 
by physical degradation can be seen as follows: note that the computation tree for a Q 
can be constructed by taking the standard computation tree of a|- a ] and independently 
erasing the observation associated to each variable leaf node with probability \a] — a. 
It follows that we can convert the computation tree of a a to that of a a _i by erasing 
all observations at the leaf nodes and by independently erasing each observation in the 
second (from the bottom) row of variable nodes with probability \a] — a. The same 
statement is true for b a . If lim^oo H(ae) = 0, i.e., if BP decoding is successful in the 
limit of large blocklengths, then the families are both complete. 



Example 1 (Density Evolution and Interpolation) Consider transmission over the 
BSC(0. 07) using a (3, 6)-regular ensemble. Fig. [H depicts the density evolution process 
for this case. This process gives rise to the sequences of densities {a^}£L , and {tv}^. 
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Figure 2: Density evolution for (3, 6)-regular ensemble over BSC(0.07). 

Fig.\^shows the interpolation of these sequences for the choices a = 1.0, 0.95, 0.9 and 0. 
and the complete such family. 
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Figure 3: Interpolation of densities. 



As a second ingredient we recall from [17] the definition of GEXIT functions. These 
GEXIT functions fulfill the Area Theorem for the case of general BMS channels. Up 
to date, GEXIT functions have been mainly used to derive upper bounds on the MAP 
threshold of iterative coding systems, see e.g., [17,19]. Here we will apply them to the 
components of LDPC ensembles. 

Definition 2 (The GEXIT Functional) Given two families of L-densities {c e } and 
{a e } parameterized by e define the GEXIT functional G(c e , a e ) by 



/oo 
a e (z)l c <(z)dz, 
-oo 



where 



l c Hz) 



oo <k. t (w) 
oo de 



logfl 



\dw 



oo dc e (w) 
oo de 



logfl + e~ w )dw 



Note that the kernel is normalized not with respect to de but with respect to dh, i.e., with 
respect to changes in the entropy. The families are required to be smooth in the sense 
that {H(c e ), G(c t , a e )} forms a piecewise continuous curve. 

Lemma 1 (GEXIT and Dual GEXIT Function) Consider a binary code C and trans- 
mission over a complete family of BMS channels characterized by their family of L- 
densities {c e }. Let {a e } denote the corresponding family of (average) extrinsic MAP den- 
sities. Then the standard GEXIT curve is given in parametric form by {H(c t ), G(c e , a e )}. 
The dual GEXIT curve is defined by {G(a e , c e ), H(a e )}. Both, standard and dual GEXIT 
curve have an area equal to r(C), the rate of the code. 



Discussion: Note that both curves are "comparable" in that they first component mea- 
sures the channel c and the second argument measure the MAP density a . The difference 
between the two lies in the choice of measure which is applied to each component. 

Proof 1 Consider the entropy H(c e * a e ). We have 



The first expression can be identified with the standard GEXIT curve except that it is 
parameterized by a generic parameter e. The second expression is essentially the same, 
but the roles of the two densities are exchanged. 

Integrate now this relationship over the whole range of e and assume that this range 
goes from "perfect" (channel) to "useless". The integral on the left clearly equals 1. To 
perform the integrals over the right reparameterize the first expression with respect to 
h = c e (w) log(l + e~ w )dw so that it becomes the standard GEXIT curve given by 

{H(c e ), G(c e , a e )}. In the same manner reparameterize the second expression by h = 
a e {w) log(l + e~ w )dw so that it becomes the curve given by {H(a t ), G(a e , c e )}. Since 
the sum of the two areas equals one and the area under the standard GEXIT curve equals 
r{C), it follows that the area under the second curve equals 1 — r{C). Finally, note that 
if we consider the inverse of the second curve by exchanging the two coordinates, i.e., 
if we consider the curve {G(a e , c e ), H(a e )}, then the area under this curve is equal to 
1 — (1 — r(C)) = r(C), as claimed. 

Example 2 (EXIT Versus GEXIT) Fig. ^compares the EXIT function to the GEXIT 
function for the [3, 1, 3] repetition code and the [6, 5, 2] single parity-check code when trans- 
mission takes place over the BSC. As we can see, the two curves are similar but distinct. 
In particular note that the areas under the GEXIT curves are equal to the rate of the 
codes but that this is not true for the EXIT functions. 

Example 3 (GEXIT Versus Dual GEXIT) Fig.\^shows the standard GEXIT func- 
tion and the dual GEXIT function for the [5, 4, 2] code and transmission over the BSC. 
Although the two curves have quite distinct shapes, the area under the two curves is the 
same. 

Lemma 2 Consider a degree distribution pair (A, p) and transmission over an BMS 
channel characterized by its L-density c so that density evolution converges to Aoo. Let 
{a Q }^L-i an d {bo,}^L denote the interpolated families as defined in Definition^ 
Then the two GEXIT curves parameterized by 




Consider now 



dH{c e *a € ) 
dt 



Using the previous representation we get 
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Figure 4: A comparison of the EXIT with the GEXIT function for the [3, 1, 3] and the 
[6, 5, 2] code. 
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Figure 5: Standard and dual GEXIT function of [5, 4, 2] code and transmission over the 
BSC. 



do not overlap and faithfully represent density evolution. Further, the area under the 
"check-node" G EXIT function is equal to 1 — J p and the area to the left of the "inverse 
dual variable node" G EXIT function is equal to H{c) JX. It follows that r(X, p) < 1 — 
H(c), i.e., the transmission rate can not exceed the Shannon limit. 

This implies that transmission approaching capacity requires a perfect matching of the 
two curves. 

Proof 2 First note that {H (a a ), G(a a , b a+ i)} is the standard GEXIT curve representing 
the action of the check nodes: a a corresponds to the density of the messages entering the 
check nodes and b a+ i represents the density of the corresponding output messages. On 
the other hand, {H(a a ),G(a a , b a )} is the inverse of the dual GEXIT curve corresponding 
to the action at the variable nodes: now the input density to the check nodes is b a and 
a a denotes the corresponding output density. 

The fact that the two curves do not overlap can be seen as follows. Fix an entropy 
value. This entropy value corresponds to a density a a for a unique value of a. The fact 
that G(a a , b a ) > G(a a , b a+ i) now follows from the fact that b a+ i -< b a and that for any 
symmetric a a this relationship stays preserved by applying the GEXIT functional. 

The statements regarding the areas of the two curves follow in a straightforward man- 
ner from the GAT and LemmaUi The bound on the achievable rate follows in the same 
manner as for the BEG: the total area of the GEXIT box equals one and the two curves 
do not overlap and have areas 1 — J p and H(c). It follows that 1 — J p + H{c) f\ < 1, 
which is equivalent to the claim r(A,p) < 1 — H(c). 

We see that the matching condition still holds even for general channels. There are a 
few important differences between the general case and the simple case of transmission 
over the BEC. For the BEC, the intermediate densities are always the BEC densities 
independent of the degree distribution. This of course enormously simplifies the task. 
Further, for the BEC, given the two EXIT curves, the progress of density evolution is 
simply given by a staircase function bounded by the two EXIT curves. For the general 
case, this staircase function still has vertical pieces but the "horizontal" pieces are in 
general at an angle. This is true since the y-axis for the "check node" step measures 
G(a ), but in the subsequent "inverse variable node" step it measures G(a a+ i, b a+ i). 

Therefore, one should think of two sets of labels on the y-axis, one measuring G(a a , b a+ i), 
and the second one measuring G(a Q+ i, b a+ i). The "horizontal" step then consists of first 
switching from the first y-axis to the second, so that the labels correspond to the same 
density b and then drawing a horizontal line until it crosses the "inverse variable node" 
GEXIT curve. The "vertical" step stays as before, i.e., it really corresponds to drawing 
a vertical line. All this is certainly best clarified by a simple example. 

Example 4 ((3, 6)-Regular Ensemble and Transmission over BSC) Consider the 
(3, 6)-regular ensemble and transmission over the BSC(0.07). The corresponding illustra- 
tions are shown in Fig. The top-left figure shows the standard GEXIT curve for the 
check node side. The top-right figure shows the dual GEXIT curve corresponding to the 
variable node side. In order to use these two curves in the same figure, it is convenient to 
consider the inverse function for the variable node side. This is shown in the bottom-left 
figure. In the bottom-right figure both curves are shown together with the "staircase" like 
function which represents density evolution. As we see, the two curves to not overlap and 
have both the correct areas. 
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Figure 6: Faithful representation of density evolution by two non-overlapping 
component-wise GEXIT functions which represent the "actions" of the check nodes and 
variable nodes, respectively. The area between the two curves equals is equal to the 
additive gap to capacity. 



As remarked earlier, one potential use of the matching condition is to find capacity 
approaching degree distribution pairs. Let us quickly outline a further such potential 
application. Assuming that we have found a sequence of capacity-achieving degree dis- 
tributions, how does the number of required iterations scale as we approach capacity. It 
has been conjectured that the the number of required iterations scales like 1/5, where 
5 is the gap to capacity. This conjecture is based on the geometric picture which the 
matching condition implies. To make things simple, imagine the two GEXIT curves as 
two parallel lines, lets say both at a 45 degree angle, a certain distance apart, and think 
of density evolution as a staircase function. From the previous results, the area between 
the lines is proportional to 5. Therefore, if we half 5 the distance between the lines has 
to be halved and one would expect that we need twice as many steps. Obviously, the 
above discussion was based on a number of simplifying assumptions. It remains to be 
seen if this conjecture can be proven rigorously. 
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